Eye tracking system

ABSTRACT

An eye tracking system for detecting position and movements of a user&#39;s eyes in a head-mounted display (HMD). The eye tracking system includes at least one eye tracking camera, an illumination source that emits infrared light towards the user&#39;s eyes, and diffraction gratings located at the eyepieces. The diffraction gratings redirect or reflect at least a portion of infrared light reflected off the user&#39;s eyes, while allowing visible light to pass. The cameras capture images of the user&#39;s eyes from the infrared light that is redirected or reflected by the diffraction gratings.

PRIORITY INFORMATION

This application is a continuation of U.S. patent application Ser. No.16/984,040, filed Aug. 3, 2020, which claims benefit of priority of U.S.Provisional Application Ser. No. 62/883,553 entitled “EYE TRACKINGSYSTEM” filed Aug. 6, 2019, which are incorporated by reference hereinin their entirety.

BACKGROUND

Virtual reality (VR) allows users to experience and/or interact with animmersive artificial environment, such that the user feels as if theywere physically in that environment. For example, virtual realitysystems may display stereoscopic scenes to users in order to create anillusion of depth, and a computer may adjust the scene content inreal-time to provide the illusion of the user moving within the scene.When the user views images through a virtual reality system, the usermay thus feel as if they are moving within the scenes from afirst-person point of view. Similarly, mixed reality (MR) combinescomputer generated information (referred to as virtual content) withreal world images or a real world view to augment, or add content to, auser's view of the world. The simulated environments of VR and/or themixed environments of MR may thus be utilized to provide an interactiveuser experience for multiple applications, such as applications that addvirtual content to a real-time view of the viewer's environment,interacting with virtual training environments, gaming, remotelycontrolling drones or other mechanical systems, viewing digital mediacontent, interacting with the Internet, or the like.

An eye tracker is a device for estimating eye positions and eyemovement. Eye tracking systems have been used in research on the visualsystem, in psychology, psycholinguistics, marketing, and as inputdevices for human-computer interaction. In the latter application,typically the intersection of a person's point of gaze with a desktopmonitor is considered.

SUMMARY

Various embodiments of methods and apparatus for eye tracking in virtualand mixed or augmented reality (VR/AR) applications are described. AVR/AR device such as a headset, helmet, goggles, or glasses (referred toherein as a head-mounted display (HMD)) is described that includes adisplay (e.g., left and right display panels) for displaying framesincluding left and right images in front of a user's eyes to thusprovide 3D virtual views to the user. The HMD may include left and righteyepieces located between the display and the user's eyes, each eyepieceincluding one or more optical lenses. The eyepieces form a virtual imageof the displayed content at a design distance which is typically closeto optical infinity of the eyepieces.

The HMD may include an eye tracking system for detecting position andmovements of the user's eyes. The eye tracking system may include atleast one eye tracking camera (e.g., infrared (IR) cameras) pointedtowards surfaces of the respective eyepieces, an illumination source(e.g., an IR light source) that emits light (e.g., IR light) towards theuser's eyes, and transmissive or reflective diffraction gratingsintegrated in the eyepieces. The diffraction gratings may, for example,be a holographic layer or film sandwiched between two optical lenses inthe eyepieces, or alternatively a holographic layer or film laminated toan image side (eye-facing) or object side (display-facing) surface of anoptical lens in the eyepieces.

In some embodiments, the light sources of the HMD emit IR light toilluminate the user's eyes. A portion of the IR light is reflected offthe user's eyes to the eye-facing surfaces of the eyepieces of the HMD.The diffraction gratings integrated in the eyepieces are configured toredirect (transmissive gratings) or reflect (reflective gratings) atleast a portion of the IR light received at the eyepieces towards the IRcameras, while allowing visible light to pass. The IR cameras, which maybe located at or near edges of the display panels when usingtransmissive gratings or alternatively at the sides of the user's face(e.g., at or near the user's cheek bones) when using reflectivegratings, capture images of the user's eyes from the infrared lightreflected or redirected by the diffraction gratings.

Integrating transmissive or reflective diffraction gratings in theeyepieces allows the spacing between the eyepieces and the displaypanels to be reduced when compared to systems that include hot mirrorslocated between the eyepieces and the display panels that reflect IRlight towards the IR cameras. Integrating reflective gratings in theeyepieces allows the user's eyes to be imaged through the eyepieceswhile improving the images (e.g., by reducing distortion) captured bythe IR cameras when compared to systems in which the IR cameras view theuser's eyes directly through the eyepieces. Integrating transmissive orreflective gratings in the eyepieces also improves the viewing angle ofthe IR cameras when compared to systems in which the IR cameras view theuser's eyes directly through the eyepieces, allowing the IR cameras toimage the user's pupils when turned away from the cameras. Integratingreflective gratings in the eyepieces allows the eye tracking cameras tobe placed at the sides of the user's face (e.g., at or near the user'scheek bones) without having to image through the eyepieces.

Images captured by the eye tracking system may be analyzed to detectposition and movements of the user's eyes, or to detect otherinformation about the eyes such as pupil dilation. For example, thepoint of gaze on the display estimated from the eye tracking images mayenable gaze-based interaction with content shown on the near-eye displayof the HMD. Other applications may include, but are not limited to,creation of eye image animations used for avatars in a VR/ARenvironment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A through 1C illustrate eye tracking systems for VR/ARhead-mounted displays (HMDs).

FIGS. 2A and 2B illustrate a VR/AR HMD that implements an eye trackingsystem that includes transmissive diffraction gratings in the eyepieces,according to some embodiments.

FIG. 3 illustrates a VR/AR HMD that implements an eye tracking systemthat includes reflective diffraction gratings in the eyepieces,according to some embodiments.

FIG. 4 illustrates an IR camera imaging a user's eye directly through aneyepiece.

FIG. 5 illustrates an IR camera imaging a user's eye through an eyepiecethat includes a transmissive diffraction grating, according to someembodiments.

FIG. 6A illustrates distortion in a system as illustrated in FIG. 4 .

FIG. 6B illustrates reduced distortion in a system as illustrated inFIG. 5 , according to some embodiments.

FIG. 7 illustrates an example assembly process for an eyepiece with anintegrated diffraction grating, according to some embodiments.

FIG. 8 illustrates example eyepieces that include diffraction gratingsat different locations in the eyepiece, according to some embodiments.

FIG. 9 shows a side view of an example HMD that implements an eyetracking system as illustrated in FIG. 2A or 2B, according to someembodiments.

FIG. 10 shows a side view of an example HMD that implements an eyetracking system as illustrated in FIG. 3 , according to someembodiments.

FIG. 11 is a block diagram illustrating components of an example VR/ARsystem that includes an eye tracking system as illustrated in FIG. 2A,2B, or 3, according to some embodiments.

FIG. 12 is a high-level flowchart illustrating a method of operation ofan HMD that includes an eye tracking system as illustrated in FIG. 2A,2B, or 3, according to some embodiments.

This specification includes references to “one embodiment” or “anembodiment.” The appearances of the phrases “in one embodiment” or “inan embodiment” do not necessarily refer to the same embodiment.Particular features, structures, or characteristics may be combined inany suitable manner consistent with this disclosure.

“Comprising.” This term is open-ended. As used in the claims, this termdoes not foreclose additional structure or steps. Consider a claim thatrecites: “An apparatus comprising one or more processor units . . . .”Such a claim does not foreclose the apparatus from including additionalcomponents (e.g., a network interface unit, graphics circuitry, etc.).

“Configured To.” Various units, circuits, or other components may bedescribed or claimed as “configured to” perform a task or tasks. In suchcontexts, “configured to” is used to connote structure by indicatingthat the units/circuits/components include structure (e.g., circuitry)that performs those task or tasks during operation. As such, theunit/circuit/component can be said to be configured to perform the taskeven when the specified unit/circuit/component is not currentlyoperational (e.g., is not on). The units/circuits/components used withthe “configured to” language include hardware—for example, circuits,memory storing program instructions executable to implement theoperation, etc. Reciting that a unit/circuit/component is “configuredto” perform one or more tasks is expressly intended not to invoke 35U.S.C. § 112, paragraph (f), for that unit/circuit/component.Additionally, “configured to” can include generic structure (e.g.,generic circuitry) that is manipulated by software or firmware (e.g., anFPGA or a general-purpose processor executing software) to operate inmanner that is capable of performing the task(s) at issue. “Configureto” may also include adapting a manufacturing process (e.g., asemiconductor fabrication facility) to fabricate devices (e.g.,integrated circuits) that are adapted to implement or perform one ormore tasks.

“First,” “Second,” etc. As used herein, these terms are used as labelsfor nouns that they precede, and do not imply any type of ordering(e.g., spatial, temporal, logical, etc.). For example, a buffer circuitmay be described herein as performing write operations for “first” and“second” values. The terms “first” and “second” do not necessarily implythat the first value must be written before the second value.

“Based On” or “Dependent On.” As used herein, these terms are used todescribe one or more factors that affect a determination. These terms donot foreclose additional factors that may affect a determination. Thatis, a determination may be solely based on those factors or based, atleast in part, on those factors. Consider the phrase “determine A basedon B.” While in this case, B is a factor that affects the determinationof A, such a phrase does not foreclose the determination of A from alsobeing based on C. In other instances, A may be determined based solelyon B.

“Or.” When used in the claims, the term “or” is used as an inclusive orand not as an exclusive or. For example, the phrase “at least one of x,y, or z” means any one of x, y, and z, as well as any combinationthereof.

DETAILED DESCRIPTION

Various embodiments of methods and apparatus for eye tracking in virtualand mixed or augmented reality (VR/AR) applications are described. AVR/AR device such as a headset, helmet, goggles, or glasses (referred toherein as a head-mounted display (HMD)) is described that includes adisplay (e.g., left and right displays) for displaying frames includingleft and right images in front of a user's eyes to thus provide 3Dvirtual views to the user. The HMD may include left and right opticallenses (referred to herein as eyepieces) located between the display andthe user's eyes. The eyepieces form a virtual image of the displayedcontent at a design distance which is typically close to opticalinfinity of the eyepieces. The HMD may include an eye tracking system(which may also be referred to as a gaze tracking system) for detectingposition and movements of the user's eyes, or for detecting otherinformation about the eyes such as pupil dilation. The point of gaze onthe display estimated from the information captured by the eye trackingsystem may, for example, allow gaze-based interaction with the contentshown on the near-eye display. Other applications may include, but arenot limited to, creation of eye image animations used for avatars in aVR/AR environment.

Embodiments of an eye tracking system for HMDs are described thatinclude at least one eye tracking camera (e.g., infrared (IR) cameras)pointed towards the surfaces of the respective eyepieces, anillumination source (e.g., an IR light source) that emits light (e.g.,IR light) towards the user's eyes, and transmissive or reflectivediffraction gratings integrated in the eyepieces (e.g., as holographicfilm). The diffraction gratings redirect or reflect light in theinfrared range while allowing visible light to pass.

In some embodiments, the diffraction grating may be implemented as aholographic film or layer sandwiched between two optical lenses of aneyepiece, or applied to an object-side or image-side surface of aneyepiece. In some embodiments, the holographic layer may be applied to asurface of one optical lens, and then the second optical lens may beattached to the holographic layer, for example using an optical couplingliquid. In some embodiments, the surfaces of the lenses between whichthe holographic layer is sandwiched may be planar. However, in someembodiments, the surfaces may be curved. Note that other types ofdiffraction gratings may be used in some embodiments. For example, insome embodiments, a photothermal reflective glass may be used as thediffraction grating. In other embodiments, a surface relief grating withmismatched index of refraction at the eye tracking wavelength may beused as the diffraction grating.

In some embodiments, the light sources of the HMD emit IR light toilluminate the user's eyes. A portion of the IR light is reflected offthe user's eyes to the eye-facing surfaces of the eyepieces of the HMD.The holographic layers integrated in the eyepieces are configured toredirect (transmissive gratings) or reflect (reflective gratings) atleast a portion of the IR light received at the eyepieces towards the IRcameras, while allowing visible light to pass. The IR cameras, which maybe located at or near edges of the display panels when usingtransmissive gratings or alternatively at the sides of the user's face(e.g., at or near the user's cheek bones) when using reflectivegratings, capture images of the user's eyes from the infrared lightreflected or redirected by the holographic layers.

Integrating transmissive or reflective gratings in the eyepiecesimproves the viewing angle of the IR cameras when compared to systems inwhich the IR cameras view the user's eyes directly through theeyepieces, allowing the IR cameras to image the user's pupils whenturned away from the cameras. Integrating transmissive or reflectivediffraction gratings in the eyepieces allows the spacing between theeyepieces and the display panels to be reduced when compared to systemsthat include hot mirrors located between the eyepieces and the displaypanels that reflect IR light towards the IR cameras. Integratingreflective gratings in the eyepieces allows the user's eyes to be imagedthrough the eyepieces while improving the images (e.g., by reducingdistortion) captured by the IR cameras when compared to systems in whichthe IR cameras view the user's eyes directly through the eyepieces.Integrating reflective gratings in the eyepieces allows the eye trackingcameras to be placed at the sides of the user's face (e.g., at or nearthe user's cheek bones) without having to image through the eyepieces.

Images captured by the eye tracking system may be analyzed to detectposition and movements of the user's eyes, or to detect otherinformation about the eyes such as pupil dilation. For example, thepoint of gaze on the display estimated from the eye tracking images mayenable gaze-based interaction with content shown on the near-eye displayof the HMD. Other applications may include, but are not limited to,creation of eye image animations used for avatars in a VR/ARenvironment.

While embodiments of an eye tracking system for HMDs are generallydescribed herein as including at least one eye tracking camerapositioned at each side of the user's face to track the gaze of both ofthe user's eyes, an eye tracking system for HMDs may also be implementedthat includes at least one eye tracking camera positioned at only oneside of the user's face to track the gaze of only one of the user'seyes.

Physical Environment

A physical environment refers to a physical world that people can senseand/or interact with without aid of electronic systems. Physicalenvironments, such as a physical park, include physical articles, suchas physical trees, physical buildings, and physical people. People candirectly sense and/or interact with the physical environment, such asthrough sight, touch, hearing, taste, and smell.

Computer-Generated Reality

In contrast, a computer-generated reality (CGR) environment refers to awholly or partially simulated environment that people sense and/orinteract with via an electronic system. In CGR, a subset of a person'sphysical motions, or representations thereof, are tracked, and, inresponse, one or more characteristics of one or more virtual objectssimulated in the CGR environment are adjusted in a manner that comportswith at least one law of physics. For example, a CGR system may detect aperson's head turning and, in response, adjust graphical content and anacoustic field presented to the person in a manner similar to how suchviews and sounds would change in a physical environment. In somesituations (e.g., for accessibility reasons), adjustments tocharacteristic(s) of virtual object(s) in a CGR environment may be madein response to representations of physical motions (e.g., vocalcommands).

A person may sense and/or interact with a CGR object using any one oftheir senses, including sight, sound, touch, taste, and smell. Forexample, a person may sense and/or interact with audio objects thatcreate 3D or spatial audio environment that provides the perception ofpoint audio sources in 3D space. In another example, audio objects mayenable audio transparency, which selectively incorporates ambient soundsfrom the physical environment with or without computer-generated audio.In some CGR environments, a person may sense and/or interact only withaudio objects.

Examples of CGR include virtual reality and mixed reality.

Virtual Reality

A virtual reality (VR) environment refers to a simulated environmentthat is designed to be based entirely on computer-generated sensoryinputs for one or more senses. A VR environment comprises a plurality ofvirtual objects with which a person may sense and/or interact. Forexample, computer-generated imagery of trees, buildings, and avatarsrepresenting people are examples of virtual objects. A person may senseand/or interact with virtual objects in the VR environment through asimulation of the person's presence within the computer-generatedenvironment, and/or through a simulation of a subset of the person'sphysical movements within the computer-generated environment.

Mixed Reality

In contrast to a VR environment, which is designed to be based entirelyon computer-generated sensory inputs, a mixed reality (MR) environmentrefers to a simulated environment that is designed to incorporatesensory inputs from the physical environment, or a representationthereof, in addition to including computer-generated sensory inputs(e.g., virtual objects). On a virtuality continuum, a mixed realityenvironment is anywhere between, but not including, a wholly physicalenvironment at one end and virtual reality environment at the other end.

In some MR environments, computer-generated sensory inputs may respondto changes in sensory inputs from the physical environment. Also, someelectronic systems for presenting an MR environment may track locationand/or orientation with respect to the physical environment to enablevirtual objects to interact with real objects (that is, physicalarticles from the physical environment or representations thereof). Forexample, a system may account for movements so that a virtual treeappears stationery with respect to the physical ground.

Examples of mixed realities include augmented reality and augmentedvirtuality.

Augmented Reality

An augmented reality (AR) environment refers to a simulated environmentin which one or more virtual objects are superimposed over a physicalenvironment, or a representation thereof. For example, an electronicsystem for presenting an AR environment may have a transparent ortranslucent display through which a person may directly view thephysical environment. The system may be configured to present virtualobjects on the transparent or translucent display, so that a person,using the system, perceives the virtual objects superimposed over thephysical environment. Alternatively, a system may have an opaque displayand one or more imaging sensors that capture images or video of thephysical environment, which are representations of the physicalenvironment. The system composites the images or video with virtualobjects, and presents the composition on the opaque display. A person,using the system, indirectly views the physical environment by way ofthe images or video of the physical environment, and perceives thevirtual objects superimposed over the physical environment. As usedherein, a video of the physical environment shown on an opaque displayis called “pass-through video,” meaning a system uses one or more imagesensor(s) to capture images of the physical environment, and uses thoseimages in presenting the AR environment on the opaque display. Furtheralternatively, a system may have a projection system that projectsvirtual objects into the physical environment, for example, as ahologram or on a physical surface, so that a person, using the system,perceives the virtual objects superimposed over the physicalenvironment.

An augmented reality environment also refers to a simulated environmentin which a representation of a physical environment is transformed bycomputer-generated sensory information. For example, in providingpass-through video, a system may transform one or more sensor images toimpose a select perspective (e.g., viewpoint) different than theperspective captured by the imaging sensors. As another example, arepresentation of a physical environment may be transformed bygraphically modifying (e.g., enlarging) portions thereof, such that themodified portion may be representative but not photorealistic versionsof the originally captured images. As a further example, arepresentation of a physical environment may be transformed bygraphically eliminating or obfuscating portions thereof.

Augmented Virtuality

An augmented virtuality (AV) environment refers to a simulatedenvironment in which a virtual or computer generated environmentincorporates one or more sensory inputs from the physical environment.The sensory inputs may be representations of one or more characteristicsof the physical environment. For example, an AV park may have virtualtrees and virtual buildings, but people with faces photorealisticallyreproduced from images taken of physical people. As another example, avirtual object may adopt a shape or color of a physical article imagedby one or more imaging sensors. As a further example, a virtual objectmay adopt shadows consistent with the position of the sun in thephysical environment.

Hardware

There are many different types of electronic systems that enable aperson to sense and/or interact with various CGR environments. Examplesinclude head mounted systems, projection-based systems, heads-updisplays (HUDs), vehicle windshields having integrated displaycapability, windows having integrated display capability, displaysformed as lenses designed to be placed on a person's eyes (e.g., similarto contact lenses), headphones/earphones, speaker arrays, input systems(e.g., wearable or handheld controllers with or without hapticfeedback), smartphones, tablets, and desktop/laptop computers. A headmounted system may have one or more speaker(s) and an integrated opaquedisplay. Alternatively, a head mounted system may be configured toaccept an external opaque display (e.g., a smartphone). The head mountedsystem may incorporate one or more imaging sensors to capture images orvideo of the physical environment, and/or one or more microphones tocapture audio of the physical environment. Rather than an opaquedisplay, a head mounted system may have a transparent or translucentdisplay. The transparent or translucent display may have a mediumthrough which light representative of images is directed to a person'seyes. The display may utilize digital light projection, OLEDs, LEDs,uLEDs, liquid crystal on silicon, laser scanning light source, or anycombination of these technologies. The medium may be an opticalwaveguide, a hologram medium, an optical combiner, an optical reflector,or any combination thereof. In one embodiment, the transparent ortranslucent display may be configured to become opaque selectively.Projection-based systems may employ retinal projection technology thatprojects graphical images onto a person's retina. Projection systemsalso may be configured to project virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface.

Eye Tracking System

FIGS. 1A through 1C illustrate eye tracking systems for VR/AR HMDs. AVR/AR HMD 100 may include a display 110 and two eyepiece lenses 120,mounted in a wearable housing. Infrared (IR) light source(s) 130 may bepositioned in the HMD 100 (e.g., around the eyepieces 120, or elsewherein the HMD 100) to illuminate the user's eyes 192 with IR light. Theuser looks through the eyepieces 120 onto the display 110. The eyepieces120 form a virtual image of the displayed content at a design distancewhich is typically close to optical infinity of the eyepieces 120. Eyetracking cameras 140 may be positioned in the HMD 100 to capture viewsof the user's eyes 192. To fit the eye tracking cameras 140 in the HMD100 while keeping the cameras 140 out of sight of the user, differentcamera optical paths have been used.

Referring to HMD 100A of FIG. 1A, the cameras 140 are positioned to havea direct view of the user's eyes. Referring to HMD 100B of FIG. 1B, thecameras 140 are positioned nearer to the display 110 such that a frontalview of the eyes 192 is captured through the eyepieces 120. Referring toHMD 100C of FIG. 1C, hot mirrors 142 are positioned between theeyepieces 120 and the display 110 to fold the camera 140 optical pathsaway from the visible light display 110 optical paths; the cameras 140may be positioned near the user's cheek bones and facing the hot mirrors142.

The camera optical paths shown in FIGS. 1A through 1C have advantagesand disadvantages. The direct view of FIG. 1A does not pass through theeyepiece, but may look onto the eye from a tilted position which maycause reduced detection accuracy of eye features at extreme gaze anglesdue to distortion, insufficient depth-of-field, and occlusions. Thethrough-the-eyepiece view of FIG. 1B allows a more centered view of theeye than the direct view of FIG. 1A, but has to deal with distortions inthe eye images introduced by the eyepiece. In addition, while thethrough-the-eyepiece view of FIG. 1B improves the viewing angle somewhatwhen compared to the direct view of FIG. 1A, this configuration stillsuffers from reduced detection accuracy of eye features at extreme gazeangles. Using hot mirrors 142 as shown in FIG. 1C may provide a centeredview of the eyes, and thus significantly improves detection accuracy ofeye features at extreme gaze angles. However, the hot mirrors 142require increased spacing between the eyepieces 120 and the display 110.

FIGS. 2A, 2B, and 3 illustrate embodiments of eye tracking system forVR/AR HMDs that include diffraction gratings in the eyepieces of theHMDs. FIGS. 2A and 2B illustrate a VR/AR HMD 200 that implements an eyetracking system that includes transmissive diffraction gratings in theeyepieces, according to some embodiments. FIG. 3 illustrates a VR/AR HMD300 that implements an eye tracking system that includes reflectivediffraction gratings in the eyepieces, according to some embodiments.Integrating transmissive or reflective gratings in the eyepiecesimproves the viewing angle of the IR cameras when compared to systems inwhich the IR cameras view the user's eyes directly as shown in FIG. 1Aor through the eyepieces as shown in FIG. 1B, allowing the IR cameras toimage the user's pupils when turned away from the cameras. Integratingtransmissive or reflective diffraction gratings in the eyepieces allowsthe spacing between the eyepieces and the display panels to be reducedwhen compared to systems as shown in FIG. 1C that include hot mirrorslocated between the eyepieces and the display panels that reflect IRlight towards the IR cameras. Integrating reflective gratings in theeyepieces as shown in FIG. 3 allows the user's eyes to be imaged throughthe eyepieces while improving the images (e.g., by reducing distortion)captured by the IR cameras when compared to systems in which the IRcameras view the user's eyes directly through the eyepieces, as shown inFIG. 1B. Integrating reflective gratings in the eyepieces as shown inFIG. 3 allows the eye tracking cameras to be placed at the sides of theuser's face (e.g., at or near the user's cheek bones) without having toimage through the eyepieces.

FIG. 2A illustrates a VR/AR HMD 200 that implements an eye trackingsystem that includes transmissive diffraction gratings in the eyepieces,according to some embodiments. VR/AR HMD 200 may include, but is notlimited to, a display 210 and two eyepieces 220, mounted in a wearablehousing or frame. Each eyepiece 220 is an optical system that mayinclude one or more optical lenses. The eye tracking system includestransmissive diffraction gratings 250 in the eyepieces 220, and at leastone eye tracking camera 240 (e.g., infrared (IR) cameras) located at ornear an edge of the display 210 (e.g., at the top, bottom, left, and/orright side of the display 210). The user looks through the eyepieces 220onto the display 210. The eyepieces 220 form a virtual image of thedisplayed content at a design distance which is typically close tooptical infinity of the eyepieces 220. The eye tracking system may, forexample, be used to track position and movement of the user's eyes 292.In some embodiments, the eye tracking system may instead or also be usedto track dilation of the user's pupils, or other characteristics of theuser's eyes 292. IR light source(s) 230 (e.g., IR LEDs) may bepositioned in the HMD 200 (e.g., around the eyepieces 220, or elsewherein the HMD 200) to illuminate the user's eyes 292 with IR light. In someembodiments, the display 210 emits light in the visible light range anddoes not emit light in the IR range, and thus does not introduce noisein the eye tracking system.

The transmissive diffraction gratings 250 are positioned at or withinthe eyepieces 220. In some embodiments, a transmissive diffractiongrating 250 may be implemented as a holographic layer 250 sandwichedbetween two optical lenses of an eyepiece 220, or as a holographic layerattached to an object-side or image-side surface of an eyepiece 220. Insome embodiments, the holographic layer 250 may be applied to a surfaceof one optical lens, and then the second optical lens may be attached tothe holographic layer 250, for example using an optical coupling liquid.The surfaces of the lenses between which the holographic layer 250 issandwiched may be, but are not necessarily, planar.

The light sources 230 of the HMD 200 emit IR light to illuminate theuser's eyes 292. A portion of the IR light is reflected off the user'seyes 292 to the eye-facing surfaces of the eyepieces 220 of the HMD 200.The transmissive holographic layers 250 integrated in the eyepieces 220are configured to redirect at least a portion of the IR light receivedat the eyepieces 220 towards the IR cameras 240, while allowing visiblelight to pass. The IR cameras 240, which may for example be located ator near an edge of the display 210, capture images of the user's eyes292 from the infrared light redirected by the transmissive holographiclayers 250.

The transmissive diffraction gratings 250 at or within the eyepieces 220allow the camera 240 optical path to be redirected, resulting in alarger incident angle of the camera axis on the center pupil location(closer to 90 degrees) than in direct-view eye tracking cameraarchitectures as shown in FIGS. 1A and 1B. The optical paths for the eyetracking cameras 240 of FIG. 2A thus provide a more direct view of theeyes 292 than the systems shown in FIGS. 1A and 1B via redirection bythe diffraction gratings 250, while allowing spacing between theeyepieces 220 and the display 210 to be reduced when compared to thesystem shown in FIG. 1C.

FIG. 2B illustrates a VR/AR HMD 200 that implements an eye trackingsystem that includes transmissive diffraction gratings in the eyepiecesand an optical prism or wedge to correct for total internal reflection(TIR), according to some embodiments. In some embodiments, the angle ofcurvature near the edge of the outer (display-facing) lens of theeyepiece 220 may result in TIR of IR light rays in that area. Tocompensate for the curvature, an optical prism or wedge 252 may belocated at the edge of the outer surface of the lens to prevent TIR ofthe IR light rays in a region near the edge of the eyepiece 220 shown inFIG. 2B.

FIG. 3 illustrates a VR/AR HMD 300 that implements an eye trackingsystem that includes reflective diffraction gratings in the eyepieces,according to some embodiments. VR/AR HMD 300 may include, but is notlimited to, a display 310 and two eyepieces 320, mounted in a wearablehousing or frame. Each eyepiece 320 is an optical system that mayinclude one or more optical lenses. The eye tracking system includesreflective diffraction gratings 360 in the eyepieces 320, and at leastone eye tracking camera 340 (e.g., infrared (IR) cameras) located at thesides of the user's face (e.g., at or near the user's cheek bones). Theuser looks through the eyepieces 320 onto the display 310. The eyepieces320 form a virtual image of the displayed content at a design distancewhich is typically close to optical infinity of the eyepieces 320. Theeye tracking system may, for example, be used to track position andmovement of the user's eyes 392. In some embodiments, the eye trackingsystem may instead or also be used to track dilation of the user'spupils, or other characteristics of the user's eyes 392. IR lightsource(s) 330 (e.g., IR LEDs) may be positioned in the HMD 300 (e.g.,around the eyepieces 320, or elsewhere in the HMD 300) to illuminate theuser's eyes 392 with IR light. In some embodiments, the display 310emits light in the visible light range and does not emit light in the IRrange, and thus does not introduce noise in the eye tracking system.

The reflective diffraction gratings 360 are positioned at or within theeyepieces 320. In some embodiments, a reflective diffraction grating 360may be implemented as a holographic layer 360 sandwiched between twooptical lenses of an eyepiece 320, or as a holographic layer attached toan object-side or image-side surface of an eyepiece 320. In someembodiments, the holographic layer 360 may be applied to a surface ofone optical lens, and then the second optical lens may be attached tothe holographic layer 360, for example using an optical coupling liquid.The surfaces of the lenses between which the holographic layer 360 issandwiched may be, but are not necessarily, planar.

The light sources 330 of the HMD 300 emit IR light to illuminate theuser's eyes 392. A portion of the IR light is reflected off the user'seyes 392 to the eye-facing surfaces of the eyepieces 320 of the HMD 300.The reflective holographic layers 360 integrated in the eyepieces 320are configured to reflect at least a portion of the IR light received atthe eyepieces 320 towards the IR cameras 340, while allowing visiblelight to pass. The IR cameras 340, which may for example be located atthe sides of the user's face (e.g., at or near the user's cheek bones),capture images of the user's eyes 392 from the infrared light reflectedby the reflective holographic layers 360.

The reflective diffraction gratings 360 at or within the eyepieces 320allow the camera 340 optical path to be folded, resulting in a largerincident angle of the camera axis on the center pupil location (closerto 90 degrees) than in direct-view eye tracking camera architectures asshown in FIGS. 1A and 1B. The optical paths for the eye tracking cameras340 of FIG. 3 thus provide a more direct view of the eyes 392 than thesystems shown in FIGS. 1A and 1B via reflection off the diffractiongratings 360, while allowing spacing between the eyepieces 320 and thedisplay 310 to be reduced when compared to the system shown in FIG. 1C.

FIG. 4 illustrates an IR camera 140 imaging a user's eye 192 directlythrough an eyepiece 120 as illustrated in FIG. 1B. Thethrough-the-eyepiece view shown in FIG. 4 allows a more centered view ofthe eye 192 than the direct view of FIG. 1A, but has to deal withdistortions in the eye images introduced by the eyepiece 120. Inaddition, while the through-the-eyepiece view shown in FIG. 4 improvesthe viewing angle somewhat when compared to the direct view of FIG. 1A,this configuration still suffers from reduced detection accuracy of eyefeatures at extreme gaze angles. FIG. 6A illustrates distortion in asystem as illustrated in FIG. 4 .

FIG. 5 illustrates an IR camera 240 imaging a user's eye 292 through aneyepiece 220 that includes a transmissive grating 250 as illustrated inFIG. 2A, according to some embodiments. Transmissive grating 250redirects IR light rays reflected off the user's eye 292 at an obliqueangle towards the IR camera 240. As can be seen in FIG. 5 , integratingthe transmissive grating 250 in the eyepiece 220 improves the viewingangle, and reduces distortion caused by the lenses of the IR camera 240when compared to a system in which the IR camera 140 views the user'seye 192 directly through the eyepiece 120 as shown in FIG. 4 , allowingthe IR camera 240 to image the user's pupil even when turned away fromthe camera 240. FIG. 6B illustrates reduced distortion in a system asillustrated in FIG. 5 when compared to a system as illustrated in FIG. 4, according to some embodiments. Integrating the transmissive grating250 in the eyepiece 220 also allows the spacing between the eyepiece 220and the display panel (not shown) to be reduced when compared to systemsas shown in FIG. 1C that include hot mirrors located between theeyepiece and the display panel that reflect IR light towards the IRcameras.

FIG. 7 illustrates an example assembly process for an eyepiece 720 withan integrated diffraction 750, according to some embodiments. Adiffraction grating 750 (e.g., a holographic film) is applied to asurface of an optical lens 721. The diffraction grating 750 is thenrecorded with transmissive or reflective holograms using a holographicrecording technology. A second optical lens 722 is attached to thediffraction grating 750, for example using an optical coupling liquid728, to produce eyepiece 720. The surfaces of the lenses 721 and 722between which the diffraction grating 750 is sandwiched may be planar,and thus the diffraction grating is planar, as shown in FIG. 7 .However, in some embodiments, the surfaces of the lenses 721 and 722between which the diffraction grating 750 is sandwiched may be curved,and the diffraction grating 750 may thus also be curved to conform tothe surfaces. Note that the shape and number of optical lenses shown ineyepiece 720 are given as an example, and are not intended to belimiting. Other shapes of optical lenses may be used, and in someembodiments one or more additional optical lenses may be attached to theoptical lenses between which the holographic layer 750 is sandwiched. Insome embodiments, a holographic layer or film may be laminated to animage side (eye-facing) or object side (display-facing) surface of aneyepiece that includes two or more optical lenses. In some embodiments,an eyepiece may include only one optical lens, and a holographic layeror film may be laminated to an image side (eye-facing) or object side(display-facing) surface of the optical lens. As mentioned, in someembodiments, diffraction grating 750 may be a holographic film. However,other types of diffraction gratings may be used in some embodiments. Forexample, in some embodiments, a photothermal reflective glass may beused as the diffraction grating. In other embodiments, a surface reliefgrating with mismatched index of refraction at the eye trackingwavelength may be used as the diffraction grating.

FIG. 8 illustrates example eyepieces that include diffraction gratingsat different locations in the eyepiece, according to some embodiments.Note that the shape and number of optical lenses shown in the eyepieces850 are given as an example, and are not intended to be limiting.Eyepiece 820A includes two optical lenses, with a diffraction grating850A located between the two lenses. Eyepiece 850B includes threeoptical lenses, with a diffraction grating 850B located between two ofthe lenses. Eyepiece 850C includes two optical lenses, with adiffraction grating 850C located at the object side surface of theeyepiece 850C. Eyepiece 850D includes two optical lenses, with adiffraction grating 850D located at the image side surface of theeyepiece 850C. Eyepiece 850E includes a single optical lens, with adiffraction grating 850E located at the image side surface of theeyepiece 850E. Eyepiece 850F includes a single optical lens, with adiffraction grating 850F located at the image side surface of theeyepiece 850F. Eyepiece 850F also illustrates a diffraction grating 850Fapplied to a curved surface of a lens.

FIG. 9 shows a side view of an example HMD 200 that implements an eyetracking system as illustrated in FIG. 2A or 2B, according to someembodiments. Note that HMD 200 as illustrated in FIG. 9 is given by wayof example, and is not intended to be limiting. In various embodiments,the shape, size, and other features of an HMD 200 may differ, and thelocations, numbers, types, and other features of the components of anHMD 200 may vary. The eye tracking system may, for example, be used totrack position and movement of the user 290's eyes 292. In someembodiments, the eye tracking system may instead or also be used totrack dilation of the user 290's pupils, or other characteristics of theuser 290's eyes 292. Information collected by the eye tracking systemmay be used in various VR or AR system functions. For example, the pointof gaze on the display 210 may be estimated from images captured by theeye tracking system; the estimated point of gaze may, for example,enable gaze-based interaction with content shown on the near-eye display210. Other applications of the eye tracking information may include, butare not limited to, creation of eye image animations used for avatars ina VR or AR environment. As another example, in some embodiments, theinformation collected by the eye tracking system may be used to adjustthe rendering of images to be projected, and/or to adjust the projectionof the images by the projection system of the HMD 200, based on thedirection and angle at which the user 290's eyes are looking. As anotherexample, in some embodiments, brightness of the projected images may bemodulated based on the user 290's pupil dilation as determined by theeye tracking system.

As shown in FIG. 9 , HMD 200 may be positioned on the user 290's headsuch that the display 210 and eyepieces 220 are disposed in front of theuser 290's eyes 292. One or more IR light source(s) 230 (e.g., IR LEDs)may be positioned in the HMD 200 (e.g., around the eyepieces 220, orelsewhere in the HMD 200) to illuminate the user 290's eyes 292 with IRlight. In some embodiments, the IR light source(s) 230 may emit light atdifferent IR wavelengths (e.g., 850 nm and 940 nm).

Each eyepiece 220 is an optical system that may include one or moreoptical lenses. The eye tracking system includes transmissivediffraction gratings 250 in the eyepieces 220, and at least one eyetracking camera 240 (e.g., an infrared (IR) cameras, for example a400×400 pixel count camera that operates at 850 nm or 940 nm, or at someother IR wavelength) located at or near an edge of the display 210(e.g., at the top, bottom, left, and/or right side of the display 210).The user looks through the eyepieces 220 onto the display 210. Theeyepieces 220 form a virtual image of the displayed content at a designdistance which is typically close to optical infinity of the eyepieces220. Note that the location and angle of eye tracking camera 240 isgiven by way of example, and is not intended to be limiting. While FIG.9 shows a single eye tracking camera 240 for each eye 292, in someembodiments there may be two or more IR cameras 240 for each eye 292.For example, in some embodiments, a camera 240 with a wider field ofview (FOV) and a camera 240 with a narrower FOV may be used for each eye292. As another example, in some embodiments, a camera 240 that operatesat one wavelength (e.g. 850 nm) and a camera 240 that operates at adifferent wavelength (e.g. 940 nm) may be used for each eye 292. Aportion of IR light emitted by light source(s) 230 reflects off the user290's eyes 292, is redirected by transmissive diffraction gratings 250to the cameras 240, and is captured by the cameras 242 to image theuser's eyes 292.

Embodiments of the HMD 200 with an eye tracking system as illustrated inFIG. 9 may, for example, be used in augmented or mixed (AR) applicationsto provide augmented or mixed reality views to the user 290. While notshown, in some embodiments, HMD 200 may include one or more sensors, forexample located on external surfaces of the HMD 200, that collectinformation about the user 290's external environment (video, depthinformation, lighting information, etc.); the sensors may provide thecollected information to a controller (not shown) of the VR/AR system.In some embodiments, the sensors may include one or more visible lightcameras (e.g., RGB video cameras) that capture video of the user'senvironment that may be used to provide the user 290 with a virtual viewof their real environment. In some embodiments, video streams of thereal environment captured by the visible light cameras may be processedby a controller of the HMD 200 to render augmented or mixed realityframes that include virtual content overlaid on the view of the realenvironment, and the rendered frames may be provided to the projectionsystem of the HMD 200 for display on display 210.

Embodiments of the HMD 200 with an eye tracking system as illustrated inFIG. 9 may also be used in virtual reality (VR) applications to provideVR views to the user 290. In these embodiments, a controller of the HMD200 may render or obtain virtual reality (VR) frames that includevirtual content, and the rendered frames may be provided to theprojection system of the HMD 200 for display on display 210.

A controller may be implemented in the HMD 200, or alternatively may beimplemented at least in part by an external device (e.g., a computingsystem) that is communicatively coupled to HMD 200 via a wired orwireless interface. The controller may include one or more of varioustypes of processors, image signal processors (ISPs), graphics processingunits (GPUs), coder/decoders (codecs), and/or other components forprocessing and rendering video and/or images. The controller may renderframes (each frame including a left and right image) that includevirtual content based at least in part on the inputs obtained from thesensors, and may provide the frames to a projection system of the HMD200 for display to display 210. FIG. 11 further illustrates componentsof a HMD and VR/AR system, according to some embodiments.

FIG. 10 shows a side view of an example HMD 300 that implements an eyetracking system as illustrated in FIG. 3 , according to someembodiments. Note that HMD 300 as illustrated in FIG. 10 is given by wayof example, and is not intended to be limiting. In various embodiments,the shape, size, and other features of an HMD 300 may differ, and thelocations, numbers, types, and other features of the components of anHMD 300 may vary. The eye tracking system may, for example, be used totrack position and movement of the user 390's eyes 392. In someembodiments, the eye tracking system may instead or also be used totrack dilation of the user 390's pupils, or other characteristics of theuser 390's eyes 392. Information collected by the eye tracking systemmay be used in various VR or AR system functions. For example, the pointof gaze on the display 310 may be estimated from images captured by theeye tracking system; the estimated point of gaze may, for example,enable gaze-based interaction with content shown on the near-eye display310. Other applications of the eye tracking information may include, butare not limited to, creation of eye image animations used for avatars ina VR or AR environment. As another example, in some embodiments, theinformation collected by the eye tracking system may be used to adjustthe rendering of images to be projected, and/or to adjust the projectionof the images by the projection system of the HMD 300, based on thedirection and angle at which the user 390's eyes are looking. As anotherexample, in some embodiments, brightness of the projected images may bemodulated based on the user 390's pupil dilation as determined by theeye tracking system.

As shown in FIG. 10 , HMD 300 may be positioned on the user 390's headsuch that the display 310 and eyepieces 320 are disposed in front of theuser 390's eyes 392. One or more IR light source(s) 330 (e.g., IR LEDs)may be positioned in the HMD 300 (e.g., around the eyepieces 320, orelsewhere in the HMD 300) to illuminate the user 390's eyes 392 with IRlight. In some embodiments, the IR light source(s) 330 may emit light atdifferent IR wavelengths (e.g., 850 nm and 940 nm).

Each eyepiece 320 is an optical system that may include one or moreoptical lenses. The eye tracking system includes reflective diffractiongratings 360 in the eyepieces 320, and at least one eye tracking camera340 (e.g., an infrared (IR) cameras, for example a 400×400 pixel countcamera that operates at 850 nm or 940 nm, or at some other IRwavelength) located at the sides of the user's face (e.g., at or nearthe user's cheek bones). The user looks through the eyepieces 320 ontothe display 310. The eyepieces 320 form a virtual image of the displayedcontent at a design distance which is typically close to opticalinfinity of the eyepieces 320. Note that the location and angle of eyetracking camera 340 is given by way of example, and is not intended tobe limiting. While FIG. 10 shows a single eye tracking camera 340 foreach eye 392, in some embodiments there may be two or more IR cameras340 for each eye 392. For example, in some embodiments, a camera 340with a wider field of view (FOV) and a camera 340 with a narrower FOVmay be used for each eye 392. As another example, in some embodiments, acamera 340 that operates at one wavelength (e.g. 850 nm) and a camera340 that operates at a different wavelength (e.g. 940 nm) may be usedfor each eye 392. A portion of IR light emitted by light source(s) 330reflects off the user 390's eyes 392, is reflected by reflectivediffraction gratings 360 to the cameras 340, and is captured by thecameras 342 to image the user's eyes 392.

Embodiments of the HMD 300 with an eye tracking system as illustrated inFIG. 10 may, for example, be used in augmented or mixed (AR)applications to provide augmented or mixed reality views to the user390. While not shown, in some embodiments, HMD 300 may include one ormore sensors, for example located on external surfaces of the HMD 300,that collect information about the user 390's external environment(video, depth information, lighting information, etc.); the sensors mayprovide the collected information to a controller (not shown) of theVR/AR system. In some embodiments, the sensors may include one or morevisible light cameras (e.g., RGB video cameras) that capture video ofthe user's environment that may be used to provide the user 390 with avirtual view of their real environment. In some embodiments, videostreams of the real environment captured by the visible light camerasmay be processed by a controller of the HMD 300 to render augmented ormixed reality frames that include virtual content overlaid on the viewof the real environment, and the rendered frames may be provided to theprojection system of the HMD 300 for display on display 310.

Embodiments of the HMD 300 with an eye tracking system as illustrated inFIG. 10 may also be used in virtual reality (VR) applications to provideVR views to the user 390. In these embodiments, a controller of the HMD300 may render or obtain virtual reality (VR) frames that includevirtual content, and the rendered frames may be provided to theprojection system of the HMD 300 for display on display 310.

A controller may be implemented in the HMD 300, or alternatively may beimplemented at least in part by an external device (e.g., a computingsystem) that is communicatively coupled to HMD 300 via a wired orwireless interface. The controller may include one or more of varioustypes of processors, image signal processors (ISPs), graphics processingunits (GPUs), coder/decoders (codecs), and/or other components forprocessing and rendering video and/or images. The controller may renderframes (each frame including a left and right image) that includevirtual content based at least in part on the inputs obtained from thesensors, and may provide the frames to a projection system of the HMD300 for display to display 310. FIG. 11 further illustrates componentsof a HMD and VR/AR system, according to some embodiments.

FIG. 11 is a block diagram illustrating components of an example VR/ARsystem 1900 that includes an eye tracking system as illustrated in FIG.2A, 2B, or 3, according to some embodiments. In some embodiments, aVR/AR system 1900 may include an HMD 2000 such as a headset, helmet,goggles, or glasses. HMD 2000 may implement any of various types ofvirtual reality projector technologies. For example, the HMD 2000 mayinclude a VR projection system that includes a projector 2020 thatdisplays frames including left and right images on screens or displays2022A and 2022B that are viewed by a user through eyepieces 2220A and2220B. The VR projection system may, for example, be a DLP (digitallight processing), LCD (liquid crystal display), or LCoS (liquid crystalon silicon) technology projection system. To create a three-dimensional(3D) effect in a 3D virtual view, objects at different depths ordistances in the two images may be shifted left or right as a functionof the triangulation of distance, with nearer objects shifted more thanmore distant objects. Note that other types of projection systems may beused in some embodiments.

In some embodiments, HMD 2000 may include a controller 2030 configuredto implement functionality of the VR/AR system 1900 and to generateframes (each frame including a left and right image) that are displayedby the projector 2020. In some embodiments, HMD 2000 may also include amemory 2032 configured to store software (code 2034) of the VR/AR systemthat is executable by the controller 2030, as well as data 2038 that maybe used by the VR/AR system 1900 when executing on the controller 2030.In some embodiments, HMD 2000 may also include one or more interfaces(e.g., a Bluetooth technology interface, USB interface, etc.) configuredto communicate with an external device 2100 via a wired or wirelessconnection. In some embodiments, at least a part of the functionalitydescribed for the controller 2030 may be implemented by the externaldevice 2100. External device 2100 may be or may include any type ofcomputing system or computing device, such as a desktop computer,notebook or laptop computer, pad or tablet device, smartphone, hand-heldcomputing device, game controller, game system, and so on.

In various embodiments, controller 2030 may be a uniprocessor systemincluding one processor, or a multiprocessor system including severalprocessors (e.g., two, four, eight, or another suitable number).Controller 2030 may include central processing units (CPUs) configuredto implement any suitable instruction set architecture, and may beconfigured to execute instructions defined in that instruction setarchitecture. For example, in various embodiments controller 2030 mayinclude general-purpose or embedded processors implementing any of avariety of instruction set architectures (ISAs), such as the x86,PowerPC, SPARC, RISC, or MIPS ISAs, or any other suitable ISA. Inmultiprocessor systems, each of the processors may commonly, but notnecessarily, implement the same ISA. Controller 2030 may employ anymicroarchitecture, including scalar, superscalar, pipelined,superpipelined, out of order, in order, speculative, non-speculative,etc., or combinations thereof. Controller 2030 may include circuitry toimplement microcoding techniques. Controller 2030 may include one ormore processing cores each configured to execute instructions.Controller 2030 may include one or more levels of caches, which mayemploy any size and any configuration (set associative, direct mapped,etc.). In some embodiments, controller 2030 may include at least onegraphics processing unit (GPU), which may include any suitable graphicsprocessing circuitry. Generally, a GPU may be configured to renderobjects to be displayed into a frame buffer (e.g., one that includespixel data for an entire frame). A GPU may include one or more graphicsprocessors that may execute graphics software to perform a part or allof the graphics operation, or hardware acceleration of certain graphicsoperations. In some embodiments, controller 2030 may include one or moreother components for processing and rendering video and/or images, forexample image signal processors (ISPs), coder/decoders (codecs), etc.

Memory 2032 may include any type of memory, such as dynamic randomaccess memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR,DDR2, DDR3, etc.) SDRAM (including mobile versions of the SDRAMs such asmDDR3, etc., or low power versions of the SDRAMs such as LPDDR2, etc.),RAIVIBUS DRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, oneor more memory devices may be coupled onto a circuit board to formmemory modules such as single inline memory modules (SIMMs), dual inlinememory modules (DIMMs), etc. Alternatively, the devices may be mountedwith an integrated circuit implementing system in a chip-on-chipconfiguration, a package-on-package configuration, or a multi-chipmodule configuration.

In some embodiments, the HMD 2000 may include one or more sensors 2050that collect information about the user's environment (video, depthinformation, lighting information, etc.). The sensors 2050 may providethe information to the controller 2030 of the VR/AR system 1900. In someembodiments, sensors 2050 may include, but are not limited to, visiblelight cameras (e.g., video cameras).

As shown in FIGS. 9 and 10 , HMD 2000 may be positioned on the user'shead such that the displays 2022A and 2022B and eyepieces 2220A and2220B are disposed in front of the user's eyes 2292A and 2292B. IR lightsources 2230A and 2230B (e.g., IR LEDs) may be positioned in the HMD2000 (e.g., around the eyepieces 2220A and 2220B, or elsewhere in theHMD 2000) to illuminate the user's eyes 2292A and 2292B with IR light.Diffraction gratings 2242A and 2242B are located at or within theeyepieces 2220A and 2220B. FIG. 11 shows transmissive diffractiongratings as illustrated in FIGS. 2A, 2B, and 9; however, reflectivediffraction gratings as shown in FIGS. 3 and 10 may be used in someembodiments. Eye tracking cameras 2240A and 2240B (e.g., IR cameras, forexample 400×400 pixel count cameras) are located at or near edges ofdisplays 2022A and 2022B, respectively. In embodiments in whichreflective diffraction gratings are used, the eye tracking cameras mayinstead be located at each side of the user's face, for example at ornear the user's cheek bones as shown in FIG. 10 . Note that the locationof eye tracking cameras 2240A and 2240B is given by way of example, andis not intended to be limiting. In some embodiments, there may be asingle eye tracking camera 2240 for each eye 2292. In some embodimentsthere may be two or more IR cameras 2240 for each eye 2292. For example,in some embodiments, a wide-angle camera 2240 and a narrower-anglecamera 2240 may be used for each eye 2292. A portion of IR light emittedby light sources 2230A and 2230B reflects off the user's eyes 2292A and2292B, is redirected (or reflected) by diffraction gratings 2242A and2242B to respective eye tracking cameras 2240A and 2240B, and iscaptured by the eye tracking cameras 2240A and 2240B to image the user'seyes 2292A and 2292B. Eye tracking information captured by the cameras2240A and 2240B may be provided to the controller 2030. The controller2030 may analyze the eye tracking information (e.g., images of theuser's eyes 2292A and 2292B) to determine eye position and movement,pupil dilation, or other characteristics of the eyes 2292A and 2292B.

The eye tracking information obtained and analyzed by the controller2030 may be used by the controller in performing various VR or AR systemfunctions. For example, the point of gaze on the displays 2022A and2022B may be estimated from images captured by the eye tracking cameras2240A and 2240B; the estimated point of gaze may, for example, enablegaze-based interaction with content shown on the displays 2022A and2022B. Other applications of the eye tracking information may include,but are not limited to, creation of eye image animations used foravatars in a VR or AR environment. As another example, in someembodiments, the information obtained from the eye tracking cameras2240A and 2240B may be used to adjust the rendering of images to beprojected, and/or to adjust the projection of the images by theprojector 2020 of the HMD 2000, based on the direction and angle atwhich the user's eyes are looking. As another example, in someembodiments, brightness of the projected images may be modulated basedon the user's pupil dilation as determined by the eye tracking system.

In some embodiments, the HMD 2000 may be configured to render anddisplay frames to provide an augmented or mixed reality (AR) view forthe user at least in part according to sensor 2050 inputs. The AR viewmay include renderings of the user's environment, including renderingsof real objects in the user's environment, based on video captured byone or more video cameras that capture high-quality, high-resolutionvideo of the user's environment for display. The AR view may alsoinclude virtual content (e.g., virtual objects, virtual tags for realobjects, avatars of the user, etc.) generated by VR/AR system 1900 andcomposited with the projected view of the user's real environment.

Embodiments of the HMD 2000 as illustrated in FIG. 11 may also be usedin virtual reality (VR) applications to provide VR views to the user. Inthese embodiments, the controller 2030 of the HMD 2000 may render orobtain virtual reality (VR) frames that include virtual content, and therendered frames may be provided to the projector 2020 of the HMD 2000for display to displays 2022A and 2022B.

FIG. 12 is a high-level flowchart illustrating a method of operation ofan HMD that includes an eye tracking system as illustrated in FIG. 2A,2B, or 3, according to some embodiments. As indicated at 3010, lightsources of the HMD emit infrared (IR) light to illuminate a user's eyes.As indicated at 3020, a portion of the IR light is reflected off theuser's eyes to diffraction gratings located at or within the eyepiecesof the HMD. For example, a diffraction grating may be implemented as aholographic layer located between two optical lenses of an eyepiece, orat an object-side or image-side surface of an eyepiece. As indicated at3030, the diffraction gratings redirect (transmissive diffractiongratings) or reflect (reflective diffraction gratings) at least aportion of the IR light towards IR cameras, while allowing visible lightto pass. As indicated at 3040, the IR cameras, located at or near edgesof the display when using transmissive diffraction gratings or locatedat the sides of the user's face (e.g., at or near the user's cheekbones) when using reflective diffraction gratings, capture images of theuser's eyes from the IR light redirected or reflected by the diffractiongratings. The arrow returning from element 3060 to element 3010indicates that the eye tracking process may be a continuous process aslong as the user is using the HMD.

The methods described herein may be implemented in software, hardware,or a combination thereof, in different embodiments. In addition, theorder of the blocks of the methods may be changed, and various elementsmay be added, reordered, combined, omitted, modified, etc. Variousmodifications and changes may be made as would be obvious to a personskilled in the art having the benefit of this disclosure. The variousembodiments described herein are meant to be illustrative and notlimiting. Many variations, modifications, additions, and improvementsare possible. Accordingly, plural instances may be provided forcomponents described herein as a single instance. Boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of claims that follow. Finally,structures and functionality presented as discrete components in theexample configurations may be implemented as a combined structure orcomponent. These and other variations, modifications, additions, andimprovements may fall within the scope of embodiments as defined in theclaims that follow.

What is claimed is:
 1. A system, comprising: a head-mounted display(HMD) configured to display visual content for viewing by a user,wherein the HMD comprises: left and right optical elements located infront of the user's left and right eyes; one or more infrared lightsources configured to emit infrared light towards the user's eyes; leftand right infrared cameras; left and right diffraction gratingsintegrated within the left and right optical elements, wherein the leftand right diffraction gratings are configured to diffract infrared lightreturned from the user's eyes towards the left and right infraredcameras, respectively; and wherein the left and right infrared camerasare configured to capture a portion of the infrared light diffracted bythe left and right diffraction gratings to generate images of the user'seyes.
 2. The system as recited in claim 1, wherein the left and rightdiffraction gratings are transmissive diffraction gratings configured toredirect infrared light returned from the user's eyes towards the leftand right infrared cameras.
 3. The system as recited in claim 1, whereinthe left and right diffraction gratings are reflective diffractiongratings configured to reflect infrared light returned from the user'seyes towards the left and right infrared cameras.
 4. The system asrecited in claim 1, wherein the left and right infrared cameras includeat least one camera that images the user's left eye and at least onecamera that images the user's right eye.
 5. The system as recited inclaim 1, wherein the diffraction gratings are implemented as aholographic film.
 6. The system as recited in claim 1, wherein thediffraction gratings are implemented as photothermal reflective glass oras a surface relief grating with mismatched index of refraction at aneye tracking wavelength.
 7. The system as recited in claim 1, furthercomprising a controller comprising one or more processors, wherein thecontroller is configured to: obtain the images of the user's eyes fromthe left and right infrared cameras; and analyze the images of theuser's eyes to determine eye tracking information.
 8. The system asrecited in claim 7, wherein the eye tracking information includes one ormore of eye position, eye movement, or pupil dilation.
 9. The system asrecited in claim 1, further comprising one or more visible light camerasconfigured to capture views of the user's environment, wherein thevisual content includes virtual content composited into the views of theuser's environment to provide an augmented or mixed reality view to theuser.
 10. The system as recited in claim 1, wherein the visual contentincludes virtual content to provide a virtual reality view to the user.11. The system as recited in claim 1, further comprising optical prismsor wedges located at edges of outer surfaces of the left and rightoptical elements to prevent total internal reflection (TIR) of infraredlight rays diffracted by the left and right diffraction gratings in aregion near the edges of the optical elements.
 12. A method, comprising:emitting, by one or more light sources of a head-mounted display (HMD),infrared (IR) light to illuminate a user's eyes; receiving, atdiffraction gratings located on a surface of or inside optical elementsof the HMD, a portion of the IR light reflected off the user's eyes;diffracting, by the diffraction gratings, at least a portion of thereceived IR light towards IR cameras of the HMD; and capturing, by theIR cameras of the HMD, images of the user's eyes from the IR lightdiffracted by the diffraction gratings.
 13. The method as recited inclaim 12, wherein the diffraction gratings are transmissive diffractiongratings, and wherein diffracting the IR light towards the IR camerascomprises redirecting the IR light towards the infrared cameras.
 14. Themethod as recited in claim 12, wherein the diffraction gratings arereflective diffraction gratings, and wherein diffracting the IR lighttowards the IR cameras comprises reflecting the IR light towards theinfrared cameras.
 15. The method as recited in claim 12, wherein theoptical elements each include one or more optical lenses, wherein thediffraction gratings are implemented as one of a holographic filmapplied to a surface of one of the one or more optical lenses, aphotothermal reflective glass attached to a surface of one of the one ormore optical lenses, or a surface relief grating on one of the one ormore optical lenses with mismatched index of refraction at the eyetracking wavelength.
 16. The method as recited in claim 12, wherein theoptical elements include optical prisms or wedges located at edges ofouter surfaces of the optical elements to prevent total internalreflection (TIR) of infrared light rays diffracted by the diffractiongratings in a region near the edges of the optical elements.
 17. Adevice, comprising: a display configured to display visual content forviewing by a user; an infrared light source configured to emit infraredlight towards the user's eye; a camera; and a transmissive diffractiongrating located on an optical path between the user's eye and thedisplay and configured to redirect the infrared light reflected from theuser's eye in a direction of the camera as the infrared light passesthrough the transmissive diffraction grating; wherein the camera isconfigured to capture a portion of the infrared light redirected by thetransmissive diffraction grating to generate images of the user's eye asilluminated by the infrared light source.
 18. The device as recited inclaim 17, wherein the optical element includes an optical prism or wedgelocated at an edge of an outer surface of the optical element to preventtotal internal reflection (TIR) of the infrared light rays redirected bythe transmissive diffraction gratings in a region near the edge of theoptical element.
 19. The device as recited in claim 17, wherein thetransmissive diffraction grating is one of a holographic film applied toa surface of an optical lens, a photothermal reflective glass attachedto a surface of an optical lens, or a surface relief grating on anoptical lens with mismatched index of refraction at the eye trackingwavelength.